Feat(cv_deploy): Expose list of targeted inactive devices if Workspace submission failed due to ResponseCode.INACTIVE_DEVICES_EXIST #4990

alexeygorbunov · 2025-02-07T04:03:39Z

Change Summary

Verify streaming status of all devices targeted by cv_deploy and raise (and update Errors) or update Warnings (depending on the force parameter of the Workspace) if any inactive device caused Workspace submission failure.

Related Issue(s)

Fixes #4038

Component(s) name

arista.avd.cv_deploy

Proposed changes

cv_client.get_inventory_devices in verify_devices_in_cloudvision_inventory already fetches actual state of all targeted devices including their streaming status. Add additional attribute _streaming to describe streaming state of CVDevices.
If Workspace submission succeeded, was forced and inactive devices were present prior to initiating submission - log warning message and append it to result.warnings as well:

TASK [arista.avd.cv_deploy : Deploy device configurations and tags to CloudVision] ***
[WARNING]: [pyavd] - Inactive devices present: [CVDevice(hostname='avd-ci-
core1', serial_number='20C292B489214DF32F9506C242A722FF',
system_mac_address='50:00:00:a1:33:1a', _exists_on_cv=True, _streaming=False)]
[WARNING]: Inactive devices present: [CVDevice(hostname='avd-ci-core1',
serial_number='20C292B489214DF32F9506C242A722FF',
system_mac_address='50:00:00:a1:33:1a', _exists_on_cv=True, _streaming=False)]

        "errors": [],
        "failed": false,
        "warnings": [
            "Inactive devices present: [CVDevice(hostname='avd-ci-core1', serial_number='20C292B489214DF32F9506C242A722FF', system_mac_address='50:00:00:a1:33:1a', _exists_on_cv=True, _streaming=False)]"
        ],
        "workspace": {
            "force": true,
            "requested_state": "submitted",
            "state": "submitted"
        }

If Workspace submission failed, was not forced, submit_result.code == ResponseCode.INACTIVE_DEVICES_EXIST and inactive devices were present prior to initiating submission - log warning message and raise exception:

TASK [arista.avd.cv_deploy : Deploy device configurations and tags to CloudVision] ***
[WARNING]: [pyavd] - Failed to submit CloudVision Workspace due to the presence
of inactive devices. Use force to override. Inactive devices:
[CVDevice(hostname='avd-ci-core1',
serial_number='20C292B489214DF32F9506C242A722FF',
system_mac_address='50:00:00:a1:33:1a', _exists_on_cv=True, _streaming=False)].

        "errors": [
            "Failed to submit CloudVision Workspace due to the presence of inactive devices. Use force to override. Inactive devices: [CVDevice(hostname='avd-ci-core1', serial_number='20C292B489214DF32F9506C242A722FF', system_mac_address='50:00:00:a1:33:1a', _exists_on_cv=True, _streaming=False)]."
        ],
        "failed": true,
        "warnings": [],
        "workspace": {
            "force": false,
            "requested_state": "submitted",
            "state": "submit failed"
        }

If Workspace submission failed, was not forced, submit_result.code == ResponseCode.INACTIVE_DEVICES_EXIST and but all devices were active prior to initiating submission - log warning message and raise exception without specifying actual devices (they might have change their streaming status between device verification and Workspace submission phases):

                msg = (
                    "Failed to submit CloudVision Workspace due to the presence of inactive devices. "
                    "Use force to override. Exact list of inactive devices is unknown."
                )
                LOGGER.warning(msg)
                raise CVWorkspaceSubmitFailedInactiveDevices(msg)

In any other failure cases - raise generic submission exception

How to test

Initiate cv_deploy molecule tests targeting CI environment
Check outputs of the cv_deploy with cv_submit_workspace_force = true test
Check outputs of the cv_deploy with cv_submit_workspace_force = false test

Checklist

User Checklist

N/A

Repository Checklist

My code has been rebased from devel before I start
I have read the CONTRIBUTING document.
My change requires a change to the documentation and documentation have been updated accordingly.
I have updated molecule CI testing accordingly. (check the box if not applicable)

github-actions · 2025-02-07T04:03:51Z

Review docs on Read the Docs

To test this pull request:

# Create virtual environment for this testing below the current directory
python -m venv test-avd-pr-4990
# Activate the virtual environment
source test-avd-pr-4990/bin/activate
# Install all requirements including PyAVD
pip install "pyavd[ansible] @ git+https://github.com/alexeygorbunov/avd.git@issue_4038_improve_inact_devs_err#subdirectory=python-avd" --force
# Point Ansible collections path to the Python virtual environment
export ANSIBLE_COLLECTIONS_PATH=$VIRTUAL_ENV/ansible_collections
# Install Ansible collection
ansible-galaxy collection install git+https://github.com/alexeygorbunov/avd.git#/ansible_collections/arista/avd/,issue_4038_improve_inact_devs_err --force
# Optional: Install AVD examples
cd test-avd-pr-4990
ansible-playbook arista.avd.install_examples

ClausHolbechArista · 2025-02-07T07:10:44Z

We should only warn if requested state for the workspace will not need to submit it. It can be useful to do builds and preview the changes even if devices are not streaming.

…by utils_zscaler.py

alexeygorbunov · 2025-02-07T19:43:43Z

We should only warn if requested state for the workspace will not need to submit it. It can be useful to do builds and preview the changes even if devices are not streaming.

fixed

    existing_devices = await verify_devices_in_cloudvision_inventory(
        ....
        raise_if_inactive=((not workspace.force) and workspace.requested_state == "submitted"),

python-avd/pyavd/_cv/client/exceptions.py

ClausHolbechArista · 2025-02-10T13:03:12Z

python-avd/pyavd/_cv/workflows/verify_devices_on_cv.py

@@ -99,6 +120,23 @@ async def verify_devices_in_cloudvision_inventory(
        device.serial_number = found_device_dict_by_hostname[device.hostname].key.device_id
        device.system_mac_address = found_device_dict_by_hostname[device.hostname].system_mac_address

+    if verify_streaming:


If feels like we are building more lists and iterating over everything again and again.

Could be just set the streaming flag during the inspection above?

You mean to extend CVDevice class to add streaming attribute?

Yeah that would be my suggestion. But if we decide to just submit no matter what, we don't even need to record this, but just set the force flag or not as given.

ClausHolbechArista · 2025-02-10T13:09:07Z

python-avd/pyavd/_cv/workflows/verify_devices_on_cv.py

+    verify_streaming: bool = False,
+    raise_if_inactive: bool = False,
+    inactive_exception_class: type = CVInactiveDevices,


I don't think we need all these extra options on this function. If you just gather the streaming state in here, you could move the invocation of the verify to the workspace submission logic where everything would make more sense. It would also mean we would not raise early (before even trying to build the workspace).

Just to confirm, after moving this streaming check over to submission phase - should we still raise (and avoid initiating submission) is we discover inactive devices targeted to configuration update when WS submission is not forced? Or should we just let submission to be initiated and let if fail (although we know in advance that it will fail)?

I know I said let's not try, but what if the device started streaming meanwhile. I think we should just submit and catch the error and provide a nice message suggesting the force option.

I think the challenge here is that in contrast to the failing build (where we can fetch details of actual ws_id + build_id pair showing what exactly failed for which device) - failure of the WS submission does not provide these details (subscribe to WorkspaceServiceStub just returns iterator of Workspace objects and they have no details about what exactly happened/failed, just an overall code and status, not telling exactly which devices were not streaming).
If we assume that streaming status can change from inactive to active between "verifying devices" step and step to submit WS - then it's probably fair to say that streaming status can change between failing WS submission and re-fetching device inventory (streaming) (if we decide to react to the failed submission postfactum). Maybe a possible way to accurately identify actual streaming state of the devices during WS submission is by subscribing to DeviceServiceStub somehow for the whole duration of the submission phase and then correlating potential changes with exact WS submission timestamp. But not sure if this is not an overkill.
Maybe we can then indeed just fetch streaming status at the initial "verification" phase and then if WS submission fails - just assume that streaming didn't change between verification and submission phases

sonarqubecloud · 2025-02-14T01:27:33Z

Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

Verify sterming status and raise before trying to submit WS

4ce3b37

alexeygorbunov requested review from a team as code owners February 7, 2025 04:03

github-actions bot assigned alexeygorbunov Feb 7, 2025

github-actions bot added the state: CI Updated CI scenario have been updated in the PR label Feb 7, 2025

alexeygorbunov marked this pull request as draft February 7, 2025 04:10

alexeygorbunov added 2 commits February 7, 2025 00:11

Adjust based on the usage of verify_devices_in_cloudvision_inventory …

c2a4f19

…by utils_zscaler.py

Relax streaming-related raise when requested WS state is not submitted

c7a6870

alexeygorbunov marked this pull request as ready for review February 7, 2025 19:46

ClausHolbechArista requested changes Feb 10, 2025

View reviewed changes

Move validation logic to WS submission stage

d8881e6

alexeygorbunov requested a review from ClausHolbechArista February 12, 2025 01:55

Merge branch 'devel' into issue_4038_improve_inact_devs_err

e20f176

alexeygorbunov marked this pull request as draft February 12, 2025 20:45

alexeygorbunov changed the title ~~Feat(cv_deploy): Verify streaming status and raise before trying to submit Workspace~~ Feat(cv_deploy): Expose list of targeted inactive devices if Workspace submission failed due to ResponseCode.INACTIVE_DEVICES_EXIST Feb 13, 2025

alexeygorbunov added 2 commits February 13, 2025 16:54

Attempt to enhance logic

a41a8ab

Change logging level when succsessfully forced with inactive

a93aef8

alexeygorbunov marked this pull request as ready for review February 14, 2025 01:26

Merge branch 'devel' into issue_4038_improve_inact_devs_err

00727c4

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Feat(cv_deploy): Expose list of targeted inactive devices if Workspace submission failed due to ResponseCode.INACTIVE_DEVICES_EXIST #4990

Feat(cv_deploy): Expose list of targeted inactive devices if Workspace submission failed due to ResponseCode.INACTIVE_DEVICES_EXIST #4990

alexeygorbunov commented Feb 7, 2025 •

edited

Loading

github-actions bot commented Feb 7, 2025

ClausHolbechArista commented Feb 7, 2025

alexeygorbunov commented Feb 7, 2025 •

edited

Loading

ClausHolbechArista Feb 10, 2025

alexeygorbunov Feb 11, 2025

ClausHolbechArista Feb 12, 2025

ClausHolbechArista Feb 10, 2025

alexeygorbunov Feb 11, 2025

ClausHolbechArista Feb 12, 2025

alexeygorbunov Feb 12, 2025

sonarqubecloud bot commented Feb 14, 2025

Feat(cv_deploy): Expose list of targeted inactive devices if Workspace submission failed due to ResponseCode.INACTIVE_DEVICES_EXIST #4990

Are you sure you want to change the base?

Feat(cv_deploy): Expose list of targeted inactive devices if Workspace submission failed due to ResponseCode.INACTIVE_DEVICES_EXIST #4990

Conversation

alexeygorbunov commented Feb 7, 2025 • edited Loading

Change Summary

Related Issue(s)

Component(s) name

Proposed changes

How to test

Checklist

User Checklist

Repository Checklist

github-actions bot commented Feb 7, 2025

ClausHolbechArista commented Feb 7, 2025

alexeygorbunov commented Feb 7, 2025 • edited Loading

ClausHolbechArista Feb 10, 2025

Choose a reason for hiding this comment

alexeygorbunov Feb 11, 2025

Choose a reason for hiding this comment

ClausHolbechArista Feb 12, 2025

Choose a reason for hiding this comment

ClausHolbechArista Feb 10, 2025

Choose a reason for hiding this comment

alexeygorbunov Feb 11, 2025

Choose a reason for hiding this comment

ClausHolbechArista Feb 12, 2025

Choose a reason for hiding this comment

alexeygorbunov Feb 12, 2025

Choose a reason for hiding this comment

sonarqubecloud bot commented Feb 14, 2025

Quality Gate passed

alexeygorbunov commented Feb 7, 2025 •

edited

Loading

alexeygorbunov commented Feb 7, 2025 •

edited

Loading